Utility of Word Embeddings in Language Identification
1. Overview
How it started
I was going through qdrant's UI after loading a dataset comprised of quranic text when I noticed something interesting, the datapoints can be distinguished by language through eye-ball inspection of a 2-D plot. qdrant's UI allows the querying of datapoints by their attributes and displaying the queried data in a 2-D plot obtained through dimensionality reduction techniques configurable by the user. The pattern was noticeable for both word representations produced using a commercially available LLM and publicly available LLM which led to the following questions:
What’s the utility of the word representations in language identification?
Can the same level of accuracy in language identification be achieved using a heuristic?
Key Findings - A heuristic approach that uses regular expressions to detect languages based on the script used in the text obtained recall and precision on language comparable to word vectors produced using nomic-embed-text-latest (nomic v1.5), nomic-embed-text-v2-moe:latest, and gemini-embedding-001. The results for nomic-embed-text-latest is especially surprising given that it is not multilingual and was developed primarily for English (en). The text used for the assessment uses script primarily associated with the language e.g. Ge’ez for Amharic, Arabic script for Arabic (ar) and this is a key factor in the outcome of the assessment. Although cross-validation on a reasonable sample (40% of entire data) has been used to ascertain consistency in results, a larger sample may reveal a different pattern. The choice of sample size was mainly been dictated by memory constraints.
So What - In situations where language is primarily expressed in the script mainly associated with it, examining the characters in the text could be more sufficient than LLM based word representations in detecting the language in the text.
- For natural language tasks such as language identification, non-commercial embedding models such as nomic-embed-text-v2-moe:latest could be as suitable as commercially available embedding models such as gemini-embedding-001.
To address the two questions (above), I adopted the following approach:
Word embeddings and their corresponding text and attributes were sampled from three collections on qdrant where each collection corresponds to vectors produced using either gemini-embedding-001, nomic-embed-text-latest, or nomic-embed-text-v2-moe:latest. The text consists of quranic script in a variety of languages obtained from tanzil.net. Details of the text data and the LLMs is provided in Section 2.0.
To enable like-for-like comparison between the vectors, samples were first collected from one collection and the text from the collection was used to obtain a matching sample for each of the other collections i.e. the only thing that differs between the three collections is their word vectors (word embeddings).
For each sample from the collections, a UMAP model was fitted on a subset of the sample i.e. the training set, obtaining a projection of the data on a 2-dimensional Riemannian manifold. PCA was first used to reduce the dimensions of the vectors before fitting the UMAP model. Subsequently a knn-classifier is fitted on the projected data, assigning a label (predicted language) to each record. This is similar to the BERTopic approach.
The performance of each umap+knn-classifier in language identification was compared to a heuristic approach that uses the count of characters in the text associated with specific languages for language identification. The heuristic uses regular expressions in counting the characters and it is described in detail in Section 6.1.
flowchart LR
A[word embeddings] -->|sampling| B[sample]
B -->|UMAP - Dimensionality Reduction| D[Projections]
D --> |KNN Classification| E[Language Classification]
B -->|Heuristic Language Classification|F[Language Classification]
F-->|Metrics Calculation|G[Results Comparison]
E-->|Metrics Calculation|G
2.0 Data
The vectorised quranic text obtained from tanzil.net is stored in the following collections on a qdrant instance running locally on a homelab cluster. Details of each collection and the vectorisers used is provided below:
A list of the languages covered in this study and whether or not they are supported by either of the embedding models is provided below
Code
Language
gemini-embedding-001
nomic v1.5 (nomic-embed-text-latest)
nomic-embed-text-v2-moe
am
Amharic
✓
✗
✓
ar
Arabic
✓
✗
✓
bs
Bosnian
✓
✗
✓
dv
Divehi
✓
✗
✗
en
English
✓
✓
✓
es
Spanish
✓
✗
✓
fa
Persian
✓
✗
✓
ha
Hausa
✓
✗
✓
nl
Dutch
✓
✗
✓
ru
Russian
✓
✗
✓
sd
Sindhi
✓
✗
✗
so
Somali
✓
✗
✓
sq
Albanian
✓
✗
✓
ta
Tamil
✓
✗
✓
tg
Tajik
✓
✗
✓
th
Thai
✓
✗
✓
tr
Turkish
✓
✗
✓
ur
Urdu
✓
✗
✓
uz
Uzbek
✓
✗
✓
zh
Chinese
✓
✗
✓
2.1 Sampling
Samples were collected from the collections starting with one collection first and then using the text in the first collection to sample records matching the same text from the other two collections.
The data is largely comprised of text in English (en) therefore some downsampling was applied to prevent skewing the results.
Code
def downsample_majority(df,samp_col="language",random_state=1235813): props = df.groupby([samp_col]).size().to_dict() sorted_props =sorted(props, key=lambda e:props[e],reverse=True) maj = sorted_props[0] avg_minority =int(np.ceil(np.mean([props[e] for e in props if e!=maj]))) A = df[df[samp_col].eq(maj)].sample(avg_minority, random_state=random_state) B = df[~df[samp_col].eq(maj)] C = pd.concat([A,B]).reset_index(drop=True)return C
An example of attributes of vectors sampled from the collection corresponding to gemini-embedding-001 is shown below. The column text in the table is what the vectors were generated from.
Code
samples[init].head()
index
id
language
name
chapter
verse
text
surah_name
revelation_order
revelation_type
revelation_note
language_description
0
236019
f2627bd6-f23a-54e5-a1a6-5bbdbad95756
en
ahmedraza
88
7
Which neither give strength nor satisfy the hu...
Al-Ghaashiya
68
Meccan
NaN
English
1
248871
ff6d1d6b-bf5d-52c7-962a-966cc590144a
en
qarai
68
26
But when they saw it, they said, ‘We have inde...
Al-Qalam
2
Medinan
Except 17-33 and 48-50, from Medina
English
2
217314
df0c4ee7-8643-57f8-86ad-e5c489867bed
en
sahih
10
77
Moses said, "Do you say [thus] about the truth...
Yunus
51
Meccan
Except 40, 94, 95, 96, from Medina
English
3
4029
042bb1ea-609d-5059-bacc-dd2fe3110639
en
qaribullah
37
63
We have made this (tree) a trial for the evild...
As-Saaffaat
56
Meccan
NaN
English
4
24268
18e67fb4-1203-54f0-bf46-d011a8464c28
en
sahih
6
95
Indeed, Allah is the cleaver of grain and date...
Al-An'aam
55
Meccan
Except 20, 23, 91, 93, 114, 151, 152, 153, fro...
English
A sample equivalent to that collected from quran-embedding-prefect-gemini-embedding-001 was collected from the other two collections detailed in Section 2.0
Code
filter_columns = ['language','name','chapter','verse','text']colls = [coll for coll in collections if coll !=init]for c in colls:if collections[c]!=None:print("Processing {}".format(c)) matching_records = fetch_records_for_dataframe( samples[init], collections[c], filter_cols = filter_columns, batch_size =200, with_vectors =True ) samp, vec = extract_payload(matching_records, exclude=[]) samples[c] = pd.DataFrame(samp) vectors[c] = vecassertlen(samples[c]) ==len(samples[init]), (f"{c}: expected {len(samples[init])} matching records, got {len(samples[c])} ""- some rows had no exact match, so the like-for-like comparison ""across collections no longer holds." )else: samples[c] =None vectors[c] =None
An example of attributes of sample records stored in quran-embedding-prefect-nomic-embed-text-latest is shown below
Code
samples['nomic-embed-text:latest'].head()
id
language
name
chapter
verse
text
surah_name
revelation_order
revelation_type
revelation_note
0
f2627bd6-f23a-54e5-a1a6-5bbdbad95756
en
ahmedraza
88
7
Which neither give strength nor satisfy the hu...
Al-Ghaashiya
68
Meccan
NaN
1
ff6d1d6b-bf5d-52c7-962a-966cc590144a
en
qarai
68
26
But when they saw it, they said, ‘We have inde...
Al-Qalam
2
Medinan
Except 17-33 and 48-50, from Medina
2
df0c4ee7-8643-57f8-86ad-e5c489867bed
en
sahih
10
77
Moses said, "Do you say [thus] about the truth...
Yunus
51
Meccan
Except 40, 94, 95, 96, from Medina
3
042bb1ea-609d-5059-bacc-dd2fe3110639
en
qaribullah
37
63
We have made this (tree) a trial for the evild...
As-Saaffaat
56
Meccan
NaN
4
18e67fb4-1203-54f0-bf46-d011a8464c28
en
sahih
6
95
Indeed, Allah is the cleaver of grain and date...
Al-An'aam
55
Meccan
Except 20, 23, 91, 93, 114, 151, 152, 153, fro...
A comparison of the statistical signature of the sample from the collection associated with gemini-embedding-001 to collections associated with the other models is shown below. The figure shows that the samples are equivalent i.e. the text, chapter, and verse match across the three samples.
Code
fig, ax = plt.subplots(3,3,figsize=(20,20), sharex=False)ax = ax.flatten()for x,c inzip([0, 3, 6],collections.keys()): samples[c].groupby('language').size().plot.bar(ax=ax[x]) ax[x].set_title(c)for x,c inzip([1,4,7],collections.keys()): samples[c].groupby('chapter').size().plot.bar(ax=ax[x]) ax[x].xaxis.set_major_locator(ticker.MaxNLocator(nbins=6)) ax[x].set_title(c)for x,c inzip([2,5,8],collections.keys()): samples[init].groupby('verse').size().plot.bar(ax=ax[x]) ax[x].xaxis.set_major_locator(ticker.MaxNLocator(nbins=6)) ax[x].set_title(init)for a in ax: a.set_ylabel("Number of Occurrences")
Code
for col in samples:if collections[col]!=None: samples[col]['language_description'] = samples[col]['language'].map(language_map)
Code
counts = {}for c in samples:if collections[c] !=None: count = samples[c].fillna('n/a').groupby(['language','language_description'],dropna=False).size() counts[c] = count
A comparison of the number of records in each sample is provided below. The sample is largely dominated by english scripts
3.0 Language Identification Using Word Representations
3.1 PCA + UMAP Parameters
The 768-dimensional vectors are first reduced with PCA before UMAP is fitted on the result - this cuts UMAP’s nearest-neighbor search cost substantially, since that cost scales with the input dimensionality. Details of the parameters used for the PCA step and the subsequent UMAP fit are provided below
umap_params =dict(n_neighbors=30, verbose=True, n_epochs=300,# a fixed seed makes the projection reproducible; UMAP then runs# single-threaded, so n_jobs is not set random_state=1235813)
3.2 KNN Classifier Parameters
The parameters used for fitting the KNeighborsClassifier is provided below
def train_test_split(df, id_col="id", train_frac=0.7):""" split a dataframe into train and test based on id_col """ tr_id = df.sample(frac=train_frac, random_state=np.random.RandomState(123456))[id_col] tst_id = df[~df[id_col].isin(tr_id)][id_col] tr_flag = df.id.isin(tr_id) tst_flag = df.id.isin(tst_id)return tr_flag, tst_flagdef train_umap(vec, pca_kwargs=None, **kwargs):""" reduce dimensionality with PCA, then fit a umap model on the reduced vectors """ dim_model = Pipeline([ ('pca', PCA(**(pca_kwargs or {}))), ('umap', UMAP(**kwargs)), ]) dim_red = dim_model.fit_transform(vec)return dim_red, dim_modeldef fit_knn_classifier(red_vec,tr_tar,**kwargs):""" fit a knn classifier """ k_class = KNeighborsClassifier(**kwargs) k_class.fit(red_vec, tr_tar)return k_classdef project(dim_model, vec, train_vec):""" project vec with the fitted PCA+UMAP model (dim_model), returning the projection and the number of points that needed a fallback. UMAP's transform embeds a point as NaN when its approximate nearest-neighbour search finds no neighbours in the training data. Such points are placed at the embedding of their exact nearest training point in PCA space instead. """ proj = dim_model.transform(vec) missing = np.isnan(proj).any(axis=1)if missing.any(): pca = dim_model.named_steps['pca'] nn = NearestNeighbors(n_neighbors=1).fit(pca.transform(train_vec)) nearest = nn.kneighbors(pca.transform(vec[missing]), return_distance=False)[:, 0] proj[missing] = dim_model.named_steps['umap'].embedding_[nearest]return proj, int(missing.sum())
Code
def compute_norm_mutual_info(df,tar_col,pred_col):""" Computes the mutual information based on columns tar_col (target) and pred_col (prediction) """ tar = df[tar_col].to_numpy() pred = df[pred_col].to_numpy() mi = normalized_mutual_info_score(tar,pred)return mi
Code
def compute_recall(df,tar_col, pred_col,average="macro"):"""Calculates the recall based on tar_col (target) and pred_col (prediction) """ tar = df[tar_col].to_numpy() pred = df[pred_col].to_numpy() rec = recall_score(tar, pred,average=average,zero_division=np.nan)return rec
Code
def compute_precision(df, tar_col,pred_col,average="macro"):""" Calculates the precision based on tar_col (target) and pred_col (prediction) """ tar = df[tar_col].to_numpy() pred = df[pred_col].to_numpy() prec = precision_score(tar,pred, average=average,zero_division=np.nan)return prec
Code
def style_dataframe_heatmap( df: pd.DataFrame, cmap: str="YlGnBu", precision: int=2 ) -> Styler:"""Returns a pandas Styler with a background heatmap pinned strictly between 0 and 1."""return ( df.style.background_gradient(cmap=cmap, vmin=0.0, vmax=1.0) .format(f"{{:.{precision}f}}") .set_properties(**{"text-align": "center"}) )
Code
def style_summary_table( df: pd.DataFrame, cmap: str="Blues", vmin: float=0.0, vmax: float=1.0,) -> Styler:"""Applies a background gradient to all rows except the first row (e.g. 'Number of Languages'). """# First row is the count row (e.g. 'Number of Languages'); the rest are stats. count_row = df.index[0] stat_rows = df.index[1:] styled = ( df.style.background_gradient( cmap=cmap, vmin=vmin, vmax=vmax, subset=pd.IndexSlice[stat_rows, :], ) .format("{:.0f}", subset=pd.IndexSlice[[count_row], :]) .format("{:.4f}", subset=pd.IndexSlice[stat_rows, :]) .set_properties(**{"text-align": "center"}) )return styled
The samples are split into a train and test set for validation purposes and a UMAP model is fitted on the portion of the sample tagged as training
Processing gemini-embedding-001
UMAP(n_epochs=300, n_jobs=1, n_neighbors=30, random_state=1235813, verbose=True)
Sun Nov 1 21:29:40 2026 Construct fuzzy simplicial set
Sun Nov 1 21:29:40 2026 Finding Nearest Neighbors
Sun Nov 1 21:29:40 2026 Building RP forest with 15 trees
Sun Nov 1 21:29:46 2026 NN descent for 15 iterations
1 / 15
2 / 15
3 / 15
Stopping threshold met -- exiting after 3 iterations
Sun Nov 1 21:30:01 2026 Finished Nearest Neighbor Search
Sun Nov 1 21:30:07 2026 Construct embedding
completed 0 / 300 epochs
completed 30 / 300 epochs
completed 60 / 300 epochs
completed 90 / 300 epochs
completed 120 / 300 epochs
completed 150 / 300 epochs
completed 180 / 300 epochs
completed 210 / 300 epochs
completed 240 / 300 epochs
completed 270 / 300 epochs
Sun Nov 1 21:31:05 2026 Finished embedding
Processing nomic-embed-text:latest
UMAP(n_epochs=300, n_jobs=1, n_neighbors=30, random_state=1235813, verbose=True)
Sun Nov 1 21:31:07 2026 Construct fuzzy simplicial set
Sun Nov 1 21:31:07 2026 Finding Nearest Neighbors
Sun Nov 1 21:31:07 2026 Building RP forest with 15 trees
Sun Nov 1 21:31:07 2026 NN descent for 15 iterations
1 / 15
2 / 15
Stopping threshold met -- exiting after 2 iterations
Sun Nov 1 21:31:12 2026 Finished Nearest Neighbor Search
Sun Nov 1 21:31:12 2026 Construct embedding
completed 0 / 300 epochs
completed 30 / 300 epochs
completed 60 / 300 epochs
completed 90 / 300 epochs
completed 120 / 300 epochs
completed 150 / 300 epochs
completed 180 / 300 epochs
completed 210 / 300 epochs
completed 240 / 300 epochs
completed 270 / 300 epochs
Sun Nov 1 21:38:06 2026 Finished embedding
Processing nomic-embed-text-v2-moe:latest
UMAP(n_epochs=300, n_jobs=1, n_neighbors=30, random_state=1235813, verbose=True)
Sun Nov 1 21:38:09 2026 Construct fuzzy simplicial set
Sun Nov 1 21:38:09 2026 Finding Nearest Neighbors
Sun Nov 1 21:38:09 2026 Building RP forest with 15 trees
Sun Nov 1 21:38:09 2026 NN descent for 15 iterations
1 / 15
2 / 15
3 / 15
4 / 15
Stopping threshold met -- exiting after 4 iterations
Sun Nov 1 21:38:15 2026 Finished Nearest Neighbor Search
Sun Nov 1 21:38:15 2026 Construct embedding
completed 0 / 300 epochs
completed 30 / 300 epochs
completed 60 / 300 epochs
completed 90 / 300 epochs
completed 120 / 300 epochs
completed 150 / 300 epochs
completed 180 / 300 epochs
completed 210 / 300 epochs
completed 240 / 300 epochs
completed 270 / 300 epochs
Sun Nov 1 21:39:27 2026 Finished embedding
/home/jovyan/work/kadamu/taleb/lib/python3.13/site-packages/umap/umap_.py:1952: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
warn(
/home/jovyan/work/kadamu/taleb/lib/python3.13/site-packages/umap/umap_.py:1952: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
warn(
/home/jovyan/work/kadamu/taleb/lib/python3.13/site-packages/umap/umap_.py:1952: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
warn(
The projected data shows distinct clusters that are comprised primarily of one main language. The colours correspond to the actual language of the text for two of the vectorisers i.e from eyeball instruction.
tar_encoder = {}for c in collections: tar_encoder[c] = LabelEncoder()# Uthmani-script rows carry language '' in the payload; relabel them "n/a" so# that '' only ever means "the heuristic could not detect a language". language_values = samples[c].language.replace('', 'n/a').fillna("n/a")# detect_language (used later for heur_lang) returns '' when no candidate# clears MIN_CONFIDENCE. Include it as a known class up front so transform()# never hits an unseen label. tar_encoder[c].fit(pd.concat([language_values, pd.Series([''])], ignore_index=True)) target = tar_encoder[c].transform(language_values) samples[c]['target_idx'] = pd.Series(target)
Code
knn_clf = {}for c in collections: knn_clf[c] = fit_knn_classifier(dim_red[c],samples[c].loc[train_flag[c],'target_idx'],**knn_params)
Code
train_pred = {}test_pred = {}for c in collections: tr_f = train_flag[c] te_f = test_flag[c] train_vec = vectors[c][tr_f, :] tr_proj, tr_missing = project(umap_model[c], train_vec, train_vec) te_proj, te_missing = project(umap_model[c], vectors[c][te_f, :], train_vec)print(f"{c}: {tr_missing} train and {te_missing} test points had no UMAP ""neighbours and were placed at their nearest training point") train_pred[c] = knn_clf[c].predict(tr_proj) test_pred[c] = knn_clf[c].predict(te_proj) samples[c]['knn_predictions'] = np.nan samples[c].loc[tr_f, 'knn_predictions'] = train_pred[c] samples[c].loc[te_f, 'knn_predictions'] = test_pred[c] samples[c]['knn_predictions'] = samples[c]['knn_predictions'].astype(int) samples[c].loc[tr_f,'Train/Test'] ="Train" samples[c].loc[te_f, 'Train/Test'] ="Test"
Sun Nov 1 21:39:31 2026 Building hub-based search tree
Sun Nov 1 21:39:39 2026 Forward diversification reduced edges from 1266540 to 421183
Sun Nov 1 21:39:41 2026 Reverse diversification reduced edges from 421183 to 421183
Sun Nov 1 21:39:43 2026 Degree pruning reduced edges from 483844 to 483844
Sun Nov 1 21:39:43 2026 Resorting data and graph based on tree order
Sun Nov 1 21:39:43 2026 Building and compiling search function
completed 0 / 100 epochs
completed 10 / 100 epochs
completed 20 / 100 epochs
completed 30 / 100 epochs
completed 40 / 100 epochs
completed 50 / 100 epochs
completed 60 / 100 epochs
completed 70 / 100 epochs
completed 80 / 100 epochs
completed 90 / 100 epochs
gemini-embedding-001: 0 train and 0 test points had no UMAP neighbours and were placed at their nearest training point
Sun Nov 1 21:39:54 2026 Building hub-based search tree
Sun Nov 1 21:39:54 2026 Forward diversification reduced edges from 1266540 to 424702
Sun Nov 1 21:39:54 2026 Reverse diversification reduced edges from 424702 to 424702
Sun Nov 1 21:39:54 2026 Degree pruning reduced edges from 527820 to 527784
Sun Nov 1 21:39:54 2026 Resorting data and graph based on tree order
Sun Nov 1 21:39:54 2026 Building and compiling search function
completed 0 / 100 epochs
completed 10 / 100 epochs
completed 20 / 100 epochs
completed 30 / 100 epochs
completed 40 / 100 epochs
completed 50 / 100 epochs
completed 60 / 100 epochs
completed 70 / 100 epochs
completed 80 / 100 epochs
completed 90 / 100 epochs
nomic-embed-text:latest: 0 train and 15 test points had no UMAP neighbours and were placed at their nearest training point
Sun Nov 1 21:40:05 2026 Building hub-based search tree
Sun Nov 1 21:40:05 2026 Forward diversification reduced edges from 1266540 to 446401
Sun Nov 1 21:40:05 2026 Reverse diversification reduced edges from 446401 to 446401
Sun Nov 1 21:40:05 2026 Degree pruning reduced edges from 538314 to 538052
Sun Nov 1 21:40:05 2026 Resorting data and graph based on tree order
Sun Nov 1 21:40:06 2026 Building and compiling search function
completed 0 / 100 epochs
completed 10 / 100 epochs
completed 20 / 100 epochs
completed 30 / 100 epochs
completed 40 / 100 epochs
completed 50 / 100 epochs
completed 60 / 100 epochs
completed 70 / 100 epochs
completed 80 / 100 epochs
completed 90 / 100 epochs
nomic-embed-text-v2-moe:latest: 0 train and 0 test points had no UMAP neighbours and were placed at their nearest training point
4.0 Results
4.1 Performance of Word Embeddings in Language Identification
The performance of the umap+knn-classification approach on all three collections was assessed using Mutual Information, Recall and Precision. Overall nomic-embed-text:latest achieved the best performance followed by gemini-embedding-001, and nomic-embed-text-v2-moe:latest. This is surprising particularly because nomic-embed-text:latest was developed primarily for English en.
- In terms of performance, the heuristic approach which is described fully in Section 6.1 achieved recall and precision comparable to nomic-embed-text:latest (v1.5), nomic-embed-text-v2-moe:latest, and gemini-embedding-001. It is however possible that a neural network trained on the embeddings would reveal a different result. It is also likely that a using a larger and more diverse dataset will show an entirely different pattern and this is a subject of further exploration. - The heuristic approach is also as informative about the languages as the LLMs. This is shown by the mutual information (MI) between the languages and the language predicted by the heuristic. - The detailed MI, recall, and precision obtained for each approach by each language is provided in Section 6.2
Code
def per_class_scores(df, encoder, tar_col="target_idx", pred_col="knn_predictions"):""" per-class recall and precision computed over the whole of df. Scoring within a single language would leave only one true class, so precision would collapse to 1/(number of distinct classes predicted). """ labels = np.arange(len(encoder.classes_)) names = encoder.classes_# '' (heuristic found no language) is never a true class, so it has no# recall and its precision is meaningless; those misses still count against# the recall of each row's true language. keep = names !='' rec = recall_score(df[tar_col], df[pred_col], labels=labels, average=None, zero_division=np.nan) prec = precision_score(df[tar_col], df[pred_col], labels=labels, average=None, zero_division=np.nan) index = pd.Index(names[keep], name="language")return pd.Series(rec[keep], index=index), pd.Series(prec[keep], index=index)agg_mi = {}agg_recall = []agg_precision = []for c in collections: samples[c]['heur_lang_prop'] = samples[c]['text'].apply(lambda e:detect_language(e)) samples[c]['heur_lang'] = samples[c]['heur_lang_prop'].apply(lambda e:e['main_language']) samples[c]['heur_lang_pred_idx'] = pd.Series(tar_encoder[c].transform(samples[c]['heur_lang']))# Score every approach on the same held-out rows. The heuristic needs no# training, but scoring it on the test rows keeps the comparison like-for-like.heur_test = samples[init][samples[init]['Train/Test'].eq("Test")]rr, pr = per_class_scores(heur_test, tar_encoder[init], pred_col='heur_lang_pred_idx')agg_mi["Heuristic - MI"] = compute_norm_mutual_info(heur_test, 'target_idx', 'heur_lang_pred_idx')agg_recall.append(rr.rename("Heuristic - Recall"))agg_precision.append(pr.rename("Heuristic - Precision"))for c in collections: test = samples[c][samples[c]['Train/Test'].eq("Test")] rk, pk = per_class_scores(test, tar_encoder[c]) agg_mi[f"UMAP+KNN - {c} - MI"] = compute_norm_mutual_info(test, 'target_idx', 'knn_predictions') agg_recall.append(rk.rename(f"UMAP+KNN - {c} - Recall")) agg_precision.append(pk.rename(f"UMAP+KNN - {c} - Precision"))
Code
# MI is a property of the whole labelling, so it is reported once per approach# rather than per language (within one language the target is constant).agg_mi_res = pd.DataFrame(agg_mi, index=["Normalised MI (test set)"])agg_rec_res = pd.concat(agg_recall,axis=1)agg_rec_desc = agg_rec_res.describe() agg_rec_desc.index = ['Number of Languages','Avg. Recall', "Std. Recall", "Minimum Recall", "25th Percentile", "50th Percentile", "75th Percentile", "Maximum Recall" ]agg_prec_res = pd.concat(agg_precision, axis=1)agg_prec_desc = agg_prec_res.describe()agg_prec_desc.index = ['Number of Languages','Avg. Precision', "Std. Precision", "Minimum Precision", "25th Percentile", "50th Percentile", "75th Percentile", "Maximum Precision" ]
Horses for Courses: In certain situations, the least sophisticated approach could be the most suitable. For instance, the heuristic approach that uses regular expressions for language detection could be more suitable for detecting languages than LLM based word representations.
5.1 Next Steps
Although the performance of the heuristic is explainable i.e. it is largely based on the scripts used in the text, it is unclear whether this is the same for the LLMs.
6.0 Appendix
### 6.1 Heuristic Language Detection
flowchart TD
A["detect_language(e)"] --> B["NFC-normalise text"]
B --> C["Count chars per script block<br/>(Latin, Arabic, Cyrillic,<br/>Ethiopic, Thaana, Tamil, Thai, CJK)"]
C --> D{"Any script<br/>chars found?"}
D -->|no| UND(["main_language = ''"]):::und
D -->|"yes: take dominant script"| E{"Script type?"}
%% ---- Stage 1: unique scripts decided outright ----
E -->|Ethiopic| U1(["am"]):::win
E -->|Thaana| U2(["dv"]):::win
E -->|Tamil| U3(["ta"]):::win
E -->|Thai| U4(["th"]):::win
E -->|CJK| U5(["zh"]):::win
%% ---- Stage 2: shared scripts need disambiguation ----
E -->|Arabic| AR["Score on EXCLUSIVE markers only"]
E -->|Cyrillic| CY["Score on EXCLUSIVE markers only"]
E -->|Latin| LA["Score on exclusive letters<br/>+ digraphs + stopwords"]
AR --> AR1{"priority check"}
AR1 -->|"Urdu retroflex / ye-barree"| ARu(["ur"])
AR1 -->|"Sindhi implosives"| ARs(["sd"])
AR1 -->|"Persian p/ch/zh/g (no ur,sd)"| ARf(["fa"])
AR1 -->|"else Arabic kaf/ye / fallback"| ARa(["ar"])
CY --> CY1{"exclusive Cyrillic marks"}
CY1 -->|"Tajik ii/uu/j"| CYt(["tg"])
CY1 -->|"Uzbek o-breve"| CYu(["uz"])
CY1 -->|"Serbian dje/lje/nje"| CYb(["bs"])
CY1 -->|"else / fallback"| CYr(["ru"])
LA --> LA1["Per-language score:<br/>char x3, digraph x1, stopword x2"]
LA1 --> LA2{"argmax<br/>(en,es,nl,sq,tr,bs,ha,so,uz)"}
LA2 --> LAout(["best Latin lang"])
%% ---- Final threshold ----
ARu & ARs & ARf & ARa & CYt & CYu & CYb & CYr & LAout --> T{"best score >=<br/>MIN_CONFIDENCE?"}
T -->|yes| WIN(["main_language = best"]):::win
T -->|no| UND
classDef win fill:#dcfce7,stroke:#16a34a,color:#14532d;
classDef und fill:#fee2e2,stroke:#dc2626,color:#7f1d1d;
Here’s what each node in the diagram represents:
A — detect_language(e): entry point; takes one text string.
B — NFC-normalise: Unicode-normalises the text so composed/decomposed forms of the same character count identically.
C — Count chars per script block: tallies how many characters fall in each script range (Latin, Arabic, Cyrillic, Ethiopic, Thaana, Tamil, Thai, CJK). This is the single measurement that drives routing.
D — Any script chars found?: guards against empty/numeric/punctuation-only input. If nothing matches, the string is unclassifiable → '' (empty string).
E — Script type?: picks the dominant script (highest count from C) and branches on it. This is the key move — decide the script once, then only consider languages that use it.
U1–U5 — am / dv / ta / th / zh: the five languages with an exclusive script (Ethiopic, Thaana, Tamil, Thai, CJK). Script alone identifies them, so they’re returned immediately with no further scoring.
AR / CY / LA — “score on exclusive markers”: the three shared-script buckets. Each scores candidate languages only on features that differ between them — never on the shared range that caused the original collisions.
AR1 (Arabic priority check) → ur / sd / fa / ar: checks distinctive letters in priority order — Urdu retroflexes/yeh-barree first, then Sindhi implosives, then Persian consonants (only if no Urdu/Sindhi marks), else Standard Arabic or fallback.
CY1 (Cyrillic marks) → tg / uz / bs / ru: routes on exclusive Cyrillic letters — Tajik (ӣ ӯ ҷ), Uzbek (ў), Serbian (ђ ј љ њ ћ џ) — defaulting to Russian when none appear.
LA1 — per-language score: for the nine Latin languages, adds up weighted evidence — exclusive letter ×3, digraph ×1, stopword ×2. Stopwords are what separate en/nl/so, which have no unique letters.
LA2 — argmax → best Latin lang: picks the highest-scoring Latin language.
T — best score ≥ MIN_CONFIDENCE?: final gate. Every shared-script branch funnels here; a winner that clears the threshold becomes main_language, otherwise the result is '' (empty string).
WIN / UND: the two terminal outcomes — a confident language code, or the unclassified bucket (main_language == '', the blank row in your recall table).